# Copyright (c) 2009-2010 Dorin Ioan Marinca (http://dorin.ioan.marinca.myopenid.com/)
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

r"""
Backlog format specification
============================

File structure
--------------
The supported backlog items are:
- story: shall indicate an outcome not an activity, what is to be done and
  how we know is done; testable; usually externally visible; INVEST:
  Independent, Negotiable, Valuable, Estimable, Small, Testable
- impediment: issue, problem, blocking point, unclear stuff
- task: shall indicate an outcome not an activity, what is to be done and how
  we know is done; usually visibility only at team level; SMART:
  Specific, Measurable, Achievable, Relevant, Time-boxed

The following links are supported between them creating multiple trees:
story (1)<---(n) story
story (1)<---(n) task
story (1)<---(n) impediment
impediment (1)<---(n) story
impediment (1)<---(n) task
impediment (1)<---(n) impediment

Also the following positional backlog items are supported:
- sprint: indicates end of backlog items committed for a sprint up to next
  occurence of sprint entry
- release: indicates a release point of all set of stories filtered by
  associated tag(s) since the last (following) release with the same tag(s) 
  (tags can represent projects, themes, categories etc)

Besides usual backlog items we can have meta-info such definitions for
specifiers & config, comment lines (starting with #), empty lines (ignored)

The backlog is a text file and has one backlog item per line.

Line structure is:

  <item_type_tag> <description> [<specifier>]...

Items are sorted by priority, lower priority first.
Timeline flows from end of file (past) to beginning of file (future).
Somewhere in the middle is the current spint tagged with +current.

Item type tags
''''''''''''''
Each line is prefixed with one character tag which identifies the item type:
+ story
! impediment [like story]
. task
\ dependency (team depends on for implementing some story)
/ release (stakeholders want it)
_ sprint
#= team member or team (pigs)
#? stakeholder or group of stakeholders (chickens)
#: state
#@ context
#+ tag
#- attribute

Description
'''''''''''
This is free text. Always put below symbols as postfix of words.
Anyway this is normal English punctuation rule. Avoid to have specifier symbols
as prefix of words (insert a blank space in between). Not applicable for
"'`([{ characters - they can prefix a word and will not be used as
specifiers.

Recommended way to express stories is: As a(n) <user>, I want <what> so
that <value,why> (ca be also '+ <what> ?<user> #<value,why>')

Specifiers
''''''''''
+AAA - assign tag, project, theme, category name to item; multiple, optional
^AAA - assign value, priority of item as integer; single, optional
:AAA - assign state of item (e.g. AAA could be wip, done, waiting or 
       canceled; empty or missing means no ongoing processing
       or focus); single, optional
=AAA - assign owner of item; multiple, optional
?AAA - assign stakeholder, origin, source of item; multiple, optional
@AAA - assign context for item (e.g. AAA could be phone, meeting, email,
       tconf, vconf, mdn, sgp, lct); multiple, optional
$NNNic[+-NNNicm][,[NNNrc[+-NNNrcm]]...] - remaining work(+-)cost modification, cost
       (initial: NNNic) or total sprint size, once started to be filled it
       should be filled daily until end of sprint; single, optional
%NNNic[+-NNNicm][,[NNNrc[+-NNNrcm]]...] - like $ but used for story points instead of
       perfect time; single, optional
&AAA - assign ID/label for an item (<-, in), other items may be part of
       current item (those marked with *AAA)
*AAA - reference to a label AAA (->, out), current item is part of item &AAA
>YYYY-MM-DD - time when item lands on backlog; single, optional
<YYYY-MM-DD - closest release date deadline (constraint); single, optional
|YYYY-MM-DD - actual release date; single, optional
#AAA - comments, external references always after description; single,
       optional

Default backlog item types & item properties
''''''''''''''''''''''''''''''''''''''''''''
- items: entries in a backlog of various types
- properties: describe an item e.g. attributes (have a value assigned), tags
      - can have different meanings when applied to different item types
Builtin items or properties are prefixed with ##.

#d# &tag &label    -regex=d+(.*)   -data=BacklogTag        -view=TubumTag      Tag for items               #builtin
#d# &attribute     -regex=d-(.*)   -data=BacklogAttr       -view=TubumAttr     Attribute for items         #builtin
d# &person         -regex=d=(.*)   -data=BacklogPerson     -view=TubumPerson   Person involved with items
d# &context        -regex=d@(.*)   -data=BacklogContext    -view=TubumContext  Context/resource for items
d# &state          -regex=d:(.*)   -data=BacklogState      -view=TubumState    State for items

d# &story          -regex=\+(.*)   -data=BacklogItem       -view=TubumItem     User story, epic or some split in between
d# &impediment     -regex=!(.*)    -data=BacklogItem       -view=TubumItem     Impediment (considered as kind of story)
d# &task           -regex=\.(.*)   -data=BacklogItem       -view=TubumItem     Task for implementing a story or impediment
d# &dependency *task -regex=\\(.*) -data=BacklogItem       -view=TubumItem     Dependency on external release (considered as kind of task)
d# &release        -regex=/(.*)    -data=BacklogRelease    -view=TubumRelease  Release
d# &sprint         -regex=_(.*)    -data=BacklogSprint     -view=TubumSprint   Sprint/Iteration

#d- &tag &+                                        -regex=.        tag associated to an item                           #builtin
#d- &attribute &-                                  -regex=.*=.*    attribute associated to an item                     #builtin
#d- &text                                          -regex=.*       text, description, content associated to an item    #builtin
#d- &rem &#                                        -regex=.*       remarks, notes, comments associated to an item      #builtin
#d- &id &&                                                         id or label that identifies an item                 #builtin
#d- &ref &dep_of &*                                                item which depends on the current one               #builtin
d- &business_value &bv &priority &prio &value &^   -regex=[0-9.]+  business value associated to an item
d- &state &:                                       -regex=.        state of an item
d- &owner &pig &responsible &=                     -regex=.        owner of an item
d- &stakeholder &chicken &?                        -regex=.        stakeholder of an item
d- &context &@                                     -regex=.        context/environment in which we deal with an item
d- &timecost &ph_cost &$                                           cost of an item in PH (perfect hours)
d- &abscost &sp_cost &%                                            cost of an item in SP (story points)
d- &t0 &>                                                          item's creation/first request time
d- &tmax &<                                                        item's deadline time
d- &tdone &|                                                       item's release time
d- &missed                                                         how many times commitment was missed? e.g. end of sprint

d- &email *person                                  -regex=.*@.*    email for any person/team
d- &reporting *person                              -regex=(daily)|(weekly)|(biweekly)|(monthly) reporting for chicken persons/teams
d- &plimit &person_limit *state                    -regex=[0-9]+   per person wip limit for states #shared by all persons
d- &tlimit &team_limit *state                      -regex=[0-9]+   per team wip limit for states #shared by all teams

d+ &unplanned                                                      known topic but which is forced in a sprint after planning/commitment
d+ &unexpected &ontheway                           +unplanned      similar with '+unplanned' but also unknown previously
d+ &srecurrent &sprint_recurrent                                   task to be created in each sprint
d+ &rrecurrent &release_recurrent                                  task to be created in each release
d+ &release &rel &major &maj &official                             official/major release
d+ &increment &inc &minor &min                                     intermediate/increment/minor release
d+ &+ &improvement                                                 adding functionality/improve
d+ &- &bugfix                                                      fixing issue/bug/regression
d+ &ok &accepted                                                   accepted by stakeholders
d+ &nok &rejected                                                  rejected by stakeholders
d+ &printed                                                        item was printed for team info radiator
d+ &default                                                        default assigned as pig or chicken if nobody assigned
d+ &stakeholder &?                                                 identify persons which are stakeholders/chickens
d+ &owner &dev &= identify                                         persons which are part of team/pigs
d+ &scrummaster &sm                                                identify persons which has scrum master role in team
d+ &productowner &po                                               identify persons which has product owner role in team

d: &waiting &suspended &blocked                                    story/task/impediment blocked by impediment #probably useless as just a dep on a impediment may mean that
d: &wip &ongoing                           -plimit=2 -tlimit=4     today someone works on this story/task/impediment
d: &done &closed &finished                                         story/task/impediment done/implemented according DoD, $ should be 0
d: &canceled &rejected &stopped                                    story/task/impediment no more needed #I think they should be cleaned in purge

Testing
-------
tubum.py --version
tubum.py -h
tubum.py .tubum\backlog.tbm --verbose
"""

from datetime import datetime
from optparse import OptionParser
from pprint import pprint
import matplotlib.pyplot as plt
import os
import re
import shlex
import sys
import time

# custom processing exceptions

class TubumError(Exception):
    name = 'Tubum'
    def __init__(self, hint):
        self.hint = hint
    def __str__(self):
        return self.name + ' error: ' + repr(self.hint)

class BacklogItemError(TubumError):
    name = 'BacklogItem'

class TubumFileError(TubumError):
    name = 'TubumFile'

class TubumCLIError(TubumError):
    name = 'TubumCommand'

# backlog primitive classes

class BacklogElementType(object):
    def __init__(self, *args, **kwargs):
        self.type = None
        self.ids = []
        self.refs = []
        self.tags = {}
        self.data_type = BacklogElement
        self.format_type = TubumBacklogElement

class BacklogElement(object):
    def __init__(self, *args, **kwargs):
        self.type = None
        self.ids = []
        self.refs = []
        self.tags = {}
        self.data_type = None
        self.format_type = None
        self.ignore = True

class BacklogElementPropertyType(object):
    def __init__(self, *args, **kwargs):
        self.name = None

# backlog element classes

# backlog

class Backlog(object):
    def __init__(self):
        self.element_types = [
                CommentBet,
                EmptyBet,
                BetBet,
                ]
        self.element_prop_types = [
                IdBept,
                RefBept,
                TagBept,
                AttrBept,
                RemBept,
                TextBept,
                BeptBept,
                ]
        self.elements = []
        self.element_props = []
        self.element_tags = {}

        self.setup()

    def filter(self, opts):
        ''' Return in place filter of backlog based on opts '''
        return self

    def is_valid(self):
        ''' Returns true if the backlog is valid '''
        return true

    def add_element_prop_type(self, ept):
        pass
    def add_element_prop(self, ep):
        pass
    def add_element_type(self, et):
        pass
    def add_element(self, e):
        pass

    def remove_element_prop_type(self, ept):
        pass
    def remove_element_prop(self, ep):
        pass
    def remove_element_type(self, et):
        pass
    def remove_element(self, e):
        pass

    def setup(self):
        ''' Initial setup of default element types and element property types '''
        pass

# tubum primtive classes

class TubumBacklogElementType(object):
    def __init__(self, *args, **kwargs):
        ''' It should be called only with keyword arguments '''
        if len(args):
            raise ('bad definition %d' % line_nmb)

        try:
            self.text = kwargs['text']
            self.ids = kwargs['ids']
            self.refs = kwargs['refs']
            self.tags = kwargs['tags']
            self.infmt = kwargs['infmt']
            self.data_type = kwargs['data_type']
        except:
            raise TubumFileError('bad definition %d' % line_nmb)

class TubumBacklogElement(object):
    def __init__(self, *args, **kwargs):
        ''' It should be called only with keyword arguments '''
        if len(args):
            raise ('bad definition %d' % line_nmb)

        try:
            self.text = kwargs['text']
            self.ids = kwargs['ids']
            self.refs = kwargs['refs']
            self.tags = kwargs['tags']
            self.infmt = kwargs['infmt']
            self.data_type = kwargs['data_type']
        except:
            raise TubumFileError('bad definition %d' % line_nmb)

class TubumBacklogElementPropertyType(object):
    def __init__(self, *args, **kwargs):
        ''' It should be called only with keyword arguments '''
        if len(args):
            raise ('bad definition %d' % line_nmb)

        try:
            self.text = kwargs['text']
            self.ids = kwargs['ids']
            self.infmt = kwargs['infmt']
            self.outfmt = kwargs['outfmt']
        except:
            raise TubumFileError('bad definition %d' % line_nmb)

class TubumBacklogElementProperty(object):
    def __init__(self, *args, **kwargs):
        ''' It should be called only with keyword arguments '''
        if len(args):
            raise ('bad definition %d' % line_nmb)

        try:
            self.text = kwargs['text']
            self.ids = kwargs['ids']
            self.infmt = kwargs['infmt']
            self.outfmt = kwargs['outfmt']
        except:
            raise TubumFileError('bad definition %d' % line_nmb)

# tubum elem & props handlers

TubumCommentBet = TubumBacklogElementType(name = 'comment',                         infmt = [re.compile(r'^#(.*)$')],     data_type = str,    ignore = True),
TubumEmptyBet   = TubumBacklogElementType(name = 'empty',                           infmt = [re.compile(r'^()$')],        data_type = None,   ignore = True),
TubumBetBet     = TubumBacklogElementType(name = 'TubumBacklogElementType',         infmt = [re.compile(r'^d (.+)$')],    data_type = TubumBacklogElementType,   ignore = True),
TubumBeptBet    = TubumBacklogElementType(name = 'TubumBacklogElementPropertyType', infmt = [re.compile(r'^d- (.+)$')],    data_type = TubumBacklogElementPropertyType,   ignore = True),

TubumIdBept     = TubumBacklogElementPropertyType(name = 'id',                      infmt = [re.compile(r'&(.+)')],         outfmt = '&%s'),
TubumRefBept    = TubumBacklogElementPropertyType(name = 'ref',                     infmt = [re.compile(r'\*(.+)')],        outfmt = '*%s'),
TubumTagBept    = TubumBacklogElementPropertyType(name = 'tag',                     infmt = [re.compile(r'\+(.+)')],        outfmt = '+%s'),
TubumAttrBept   = TubumBacklogElementPropertyType(name = 'attribute',               infmt = [re.compile(r'-(\w+)=(.+)')],   outfmt = '-%s=%s'),
TubumRemBept    = TubumBacklogElementPropertyType(name = 'rem',                     infmt = [re.compile(r'#(.+)')],         outfmt = '#%s'),
TubumTextBept   = TubumBacklogElementPropertyType(name = 'text',                    infmt = [re.compile(r'(.+)')],          outfmt = '%s'),
            
# tubum

class TubumFile(object):
    def __init__(self, file):
        ''' Assumption: tokens do not contain blank spaces '''
        self.re_bspace = re.compile(r'[ \t]+')
        self.element_types = [
                TubumCommentBet,
                TubumEmptyBet,
                TubumBetBet,
                ]
        self.element_prop_types = [
                TubumIdBept,
                TubumRefBept,
                TubumTagBept,
                TubumAttrBept,
                TubumRemBept,
                TubumTextBept,
                TubumBeptBept,
                ]
        self.elements = []
        self.element_props = []
        # get the file content
        self.content = file.read()
        # create an empty backlog that will be popolated by self.parse()
        self.backlog = Backlog()

    def parse_token(self, backlog, token, state, opts):
        if opts.debug:
            print 'token:', token

        for tokname, tokregex in self.element_prop_types:
            if opts.debug:
                print ' token? %s' % tokname
            for tr in tokregex:
                m = tr.match(token)
                if m:
                    if opts.debug:
                        print '  found %s: %s' % (tokname, m.groups(1))
                    return


    def parse_line(self, line, line_nmb, opts):
        ''' Identify the item type and create an object for it '''

        # TODO: each e* list to be parsed
        # each line must match one and only one backlog element type
        for e in self.element_types:
            for infmt in e.infmt:
                m = infmt.match(line)
                if m:
                    bi = BacklogElement()
                    bi.type = tname
                    bi.line_nmb = line_nmb

                    tok_lst = re.split(self.re_bspace, m.group(1))
                    tok_state = 'description'
                    for tok in tok_lst:
                        if len(tok) > 0:
                            # parse token
                            self.parse_token(bi, tok, tok_state, opts)

                    if opts.debug:
                        pprint(bi.__dict__)
                    self.backlog.items.append(bi)
                    return

        raise TubumFileError('unknown element type on line %d' % line_nmb)

    def parse(self, opts):
        self.nmb_lines = 0
        for line in self.content.splitlines(False):
            self.nmb_lines += 1
            self.parse_line(line, self.nmb_lines, opts)
        return self.backlog

class TubumCommand(object):
    ''' command name '''
    def __init__(self):
        self.name = self.__doc__
        self.usage = '%-10s           %s' % (self.__doc__, self.process.__doc__) + os.linesep

    def process(self, opts = None, args = None):
        ''' command description '''
        pass

class TubumCommandVersion(TubumCommand):
    '''version'''

    def process(self, opts, args):
        ''' Print version '''
        print 'tubum-0.0'
        print 'Copyright (c) 2009-2010 Dorin Ioan Marinca (http://dorin.ioan.marinca.myopenid.com/)'

class TubumCommandVerify(TubumCommand):
    '''verify'''

    def process(self, opts, args):
        ''' Verify correctness of the backlog FILE in tubum format '''
        pass

class TubumCommandExport(TubumCommand):
    '''export'''

    def process(self, opts, args):
        ''' Export selected items from backlog FILE in tubum format '''
        pass

class TubumCommandReport(TubumCommand):
    '''report'''

    def process(self, opts, args):
        ''' Generate reports with formatted and filtered items from backlog FILE producing also statistical data, forecasts or charts '''
        pass

class TubumCLI(object):
    def __init__(self):
        # cli commands
        self.cmds = [
                TubumCommandVersion(),
                TubumCommandVerify(),
                TubumCommandExport(),
                TubumCommandReport(),
                ]
        self.cmd_process = {}
        for _cmd in self.cmds:
            self.cmd_process[_cmd.__doc__] = _cmd.process

        usage =   'usage: %prog command [options] FILE' + os.linesep \
                + os.linesep \
                + 'Commands:' + os.linesep

        for cmd in self.cmds:
            usage += '  ' + cmd.usage

        self.default_lsel = []
        self.default_lnosel = []
        #self.default_sort = ['_', ':', '^', '$', '=']
        self.default_sort = []
        self.default_csel = ['_',':','^','$','=']
        self.default_cfmt = [
                    'k%s',
                    '/%s',
                    '_%s',
                    'd%s',
                    '^%s',
                    '+%s',
                    ':%s',
                    '$%s',
                    '=%s',
                    '?%s',
                    '&%s',
                    '*%s',
                    '@%s',
                    '#%s',
                    '<%s',
                    '>%s',
                    '|%s',
                    'i%g',
                    'e%g',
                    'a%g',
                    'r%g',
                    't%g',
                    'l%d',
                    'L%s',
                    ]

        self.parser = OptionParser(usage = usage)

        self.parser.add_option('-I', '-l', '--grep', '--line-filter-in', dest = 'lsel',
                help = 'where LSEL format is "[<specifier>]<substring>". Items having pattern <substring> in <specifier> will be processed, others will be ignored; default LSEL = %s' % repr(self.default_lsel),
                action = 'append', metavar = 'LSEL', default = [])
        self.parser.add_option('-X', '-L', '--ngrep', '--line-filter-out', dest = 'lnosel',
                help = 'where LSEL format is "[<specifier>]<substring>". Items not having pattern <substring> in <specifier> will be processed, others will be ignored; default LNOSEL = %s' % repr(self.default_lnosel),
                action = 'append', metavar = 'LSEL', default = [])
        self.parser.add_option('-s', '--sort', dest = 'sort',
                help = 'where SORT format is "<specifier>". Set sort specification to SORT pattern: field <specifier> will be sorted in ascending order, first specified index has highest priority; default PATTERN = %s' % repr(self.default_sort),
                action = 'append', metavar = 'SORT', default = [])
        self.parser.add_option('-c', '--field', '--column-filter-in', dest = 'csel',
                help = 'where CSEL format is "<specifier>". Only specified item attibutes are processed, others will be ignored; default CSEL = %s' % repr(self.default_csel),
                action = 'append', metavar = 'CSEL', default = [])
        self.parser.add_option('-f', '--format', '--column-format', dest = 'cfmt',
                help = 'where CFMT format is "<specifier>%%<printf-format>". The format used for printing is specified for each item attribute; default CFMT = %s' % repr(self.default_cfmt),
                action = 'append', metavar = 'CFMT', default = [])
        self.parser.add_option('-a', '--analysis', dest = 'analysis',
                help = 'print stats, forecast in filtered scope',
                action = 'store_true', default = False)
        self.parser.add_option('-b', '--bdc', '--burndown-chart', dest = 'bdchart',
                help = 'print a burndown chart in filtered scope',
                action = 'store_true', default = False)
        self.parser.add_option('-x', '--bdcsize', '--burndown-chart-size', dest = 'bdchart_size',
                help = 'set size (WIDTHxHEIGHT) for the burndown chart',
                metavar = 'WIDTHxHEIGHT', default = '60x20')
        self.parser.add_option('-r', '--run-as-of-date', dest = 'run_aod',
                help = 'execute program as of date RAOD (YYYY-MM-DD); default RAOD = today',
                metavar = 'RAOD', action = 'store', default = datetime.now().strftime('%Y-%m-%d'))
        self.parser.add_option('-o', '--output', dest = 'output',
                help = 'set output file',
                action = 'store', default = None)
        self.parser.add_option('-v', '--verbose', dest = 'verbose',
                help = 'generate extra details in output for helping to understand execution steps and data presentation',
                action = 'store_true', default = False)
        self.parser.add_option('-d', '--debug', dest = 'debug',
                help = 'generate extra details in output for debug purposes',
                action = 'store_true', default = False)
        self.parser.add_option('--version', dest = 'version',
                help = 'print version of program',
                action = 'store_true', default = False)

        (self.options, self.args) = self.parser.parse_args()

        #replace default values of some options with those specified in cmd line
        if len(self.options.lsel) == 0:
            self.options.lsel = self.default_lsel
        if len(self.options.lnosel) == 0:
            self.options.lnosel = self.default_lnosel
        if len(self.options.sort) == 0:
            self.options.sort = self.default_sort
        if len(self.options.csel) == 0:
            self.options.csel = self.default_csel
        if len(self.options.cfmt) == 0:
            self.options.cfmt = self.default_cfmt

    def process(self):
        ''' Process the CLI commands '''

        if self.options.version:
            self.cmd_process['version'](self.options, self.args)
            sys.exit(0)

        if len(self.args) == 1:
            # set default command to be verify
            if self.options.verbose:
                print 'verify command will be processed by default.'
            self.args = ['verify', self.args[0]]
        elif len(self.args) == 0 or len(self.args) > 2:
            raise TubumCLIError('bad number of args')

        if self.options.debug:
            print 'args:'
            pprint(self.args)
            print 'options:'
            pprint(self.options)

        # read backlog file
        f = file(self.args[1], 'r')
        self.tbf = TubumFile(f)
        f.close()

        # parse backlog file & create the backlog data (original content)
        self.backlog = self.tbf.parse(self.options)

        # keep for further processing only selected elements filtered according lsel
        self.backlog = self.backlog.filter(self.options)

        # cmd dispatcher
        for cmd in self.cmds:
            if self.args[0] == cmd.name:

                if self.options.output:
                    self.fout = open(self.options.output, 'w')
                else:
                    self.fout = sys.stdout

                # process will apply only on filtered items
                self.current_cmd = cmd
                self.current_cmd.process()

                self.fout.close()

                sys.exit(0)

        raise TubumCLIError('process: unknown command in args')

# MAIN -----------------------------------------------------------------------
if __name__ == '__main__':
    tbmcli = TubumCLI()
    tbmcli.process()

# vim:tw=0
